Cluster-based dependency signaling

ABSTRACT

The signalization of the inter-layer dependencies between layers of a multi-layered data stream is described. A good compromise between a too intensive restriction of the potential diversity of inter-layer dependencies on the one hand and a too complex signaling of the inter-layer dependencies on the other hand has been found by describing the inter-layer dependencies by way of a first inter-dependency syntax structure indicating inter-dependencies between pairs of different values representable by a base layer-ID and a second inter-dependency syntax structure indicating inter-dependencies between pairs of different values representable by an extension layer-ID, the base layer ID and extension layer ID indexing the layers the portions of the multi-layer data stream are associated with. In accordance with this concept, emphasis may be shifted between increased diversity of the signalizable inter-layer dependencies on the one hand and reduced side-information overhead for signaling the inter-layer dependencies on the other hand.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/795,655 filed Feb. 20, 2020 which is a continuation of U.S.patent application Ser. No. 16/460,230 filed Jul. 2, 2019, now U.S. Pat.No. 10,609,399, which is a continuation of U.S. patent application Ser.No. 16/157,723 filed Oct. 11, 2018, now U.S. Pat. No. 10,425,651, whichis a continuation of U.S. patent application Ser. No. 14/995,222 filedJan. 14, 2016, now U.S. Pat. No. 10,142,639, which is a continuation ofInternational Application PCT/EP2014/065181 filed Jul. 15, 2014, andclaims priority from U.S. Patent Application 61/846,479 filed Jul. 15,2013, all of which are incorporated herein by reference theirentireties.

The present application is concerned with cluster-based dependencysignaling such as, for example, in the field of multi-layered videocoding.

BACKGROUND OF THE INVENTION

Video codecs specifications such as the HEVC base specification [1] andits extensions such as the multi-view extension [2] [3] usually inheritmechanisms to signal the layer to which each coded chunk of video databelongs to in a layered coding scenario. The signaling mechanism needsto be readily available, cp. the layer identifier in den NAL unit header(nuh_layer_id) in HEVC as given in the table in FIG. 1 .

With its finite size in terms of bits, the respective HEVC syntaxelement allows for 64 values to be used to identify scalable layers,views and/or depth. For multi-view video using the current HEVC NAL unitheader syntax, this for example implies a constraint of 64 views or 32view plus depth combinations as maximum. While this is sufficient formany common stereo view scenarios and other applications, huge cameraarrays with more than 100 views for applications like light fieldimaging [4], [5] or holographic displays [6] may use an extendablemechanism for signaling a larger space of layer identifiers.

Furthermore, structuring the space of layer identifiers by groupingcertain layers or views with respect to their coding dependencies orspatial relation to each other may be of benefit. Further, providing acompact representation of the coding dependencies in the structuredlayer identifier space is also of benefit as deriving such informationfrom already available information within the coded video bitstream mayinvolve intolerable computational resources.

SUMMARY

According to an embodiment, a device may be configured to process amulti-layered data stream into which a video material is coded atdifferent layers, corresponding to different levels of informationamount, using inter-layer prediction, the multi-layer data streamincluding a plurality of packets, each of which is associated with oneof the different layers, each layer being indexed by a base layer ID ora base layer ID and an extension layer ID, and the device may beconfigured to: read, from the multi-layered data stream, a firstinter-dependency syntax structure indicating interdependencies betweenpairs of different values representable by the base layer-ID, and asecond inter-dependency syntax structure indicating inter-dependenciesbetween pairs of different values representable by the extensionlayer-ID, and construct, based on the first and second inter-dependencysyntax structures, an inter-layer dependency description revealingpossible inter-layer prediction dependencies between the differentlayers.

According to another embodiment, a device may be configured to process amulti-layered data stream into which a video material is coded atdifferent layers, corresponding to different levels of informationamount, using inter-layer prediction, the multi-layer data streamincluding a plurality of packets, each of which is associated with oneof the different layers, each layer being indexed by a base layer ID ora base layer ID and an extension layer ID, and the device may beconfigured to: read, from the multi-layered data stream, a firstinter-dependency syntax structure indicating interdependencies betweenpairs of different values representable by the base layer-ID, and asecond inter-dependency syntax structure indicating inter-dependenciesbetween pairs of different values representable by the extensionlayer-ID, and construct, based on the first and second inter-dependencysyntax structures, an inter-layer dependency description revealingpossible inter-layer prediction dependencies between the differentlayers.

According to another embodiment, an encoder may be configured to encodeinto a multi-layered data stream a video material at different layers,corresponding to different levels of information amount, usinginter-layer prediction, the multi-layer data stream including aplurality of packets, each of which is associated with one of thedifferent layers, each layer being indexed by a base layer ID or a baselayer ID and an extension layer ID, and may be configured to: insert,into the multi-layered data stream, a first inter-dependency syntaxstructure indicating interdependencies between pairs of different valuesrepresentable by the base layer-ID, and a second inter-dependency syntaxstructure indicating inter-dependencies between pairs of differentvalues representable by the extension layer-ID, with setting the firstand second inter-dependency syntax structures such that, based on thefirst and second inter-dependency syntax structures, an inter-layerdependency description revealing possible inter-layer predictiondependencies between the different layers is constructible.

According to another embodiment, a method for encoding into amulti-layered data stream a video material at different layers,corresponding to different levels of information amount, usinginter-layer prediction, the multi-layer data stream including aplurality of packets, each of which is associated with one of thedifferent layers, each layer being indexed by a base layer ID or a baselayer ID and an extension layer ID, may have the steps of: inserting,into the multi-layered data stream, a first inter-dependency syntaxstructure indicating interdependencies between pairs of different valuesrepresentable by the base layer-ID, and a second inter-dependency syntaxstructure indicating inter-dependencies between pairs of differentvalues representable by the extension layer-ID, with setting the firstand second inter-dependency syntax structures such that, based on thefirst and second inter-dependency syntax structures, an inter-layerdependency description revealing possible inter-layer predictiondependencies between the different layers is constructible.

According to another embodiment, a non-transitory digital storage mediummay have computer-readable code stored thereon to perform, when saidstorage medium is run by a computer, the inventive methods.

One of the aspects of the present application is concerned with thesignalization of the inter-layer dependencies between layers of amulti-layered data stream. According to this aspect, a good compromisebetween a too intensive restriction of the potential diversity ofinter-layer dependencies on the one hand and a too complex signaling ofthe inter-layer dependencies on the other hand has been found bydescribing the inter-layer dependencies by way of a firstinter-dependency syntax structure indicating inter-dependencies betweenpairs of different values representable by a base layer-ID and a secondinter-dependency syntax structure indicating inter-dependencies betweenpairs of different values representable by an extension layer-ID, thebase layer ID and extension layer ID indexing the layers the portions ofthe multi-layer data stream are associated with. In accordance with thisconcept, emphasis may be shifted between increased diversity of thesignalizable inter-layer dependencies on the one hand and reducedside-information overhead for signaling the inter-layer dependencies onthe other hand: for example, calling the sets of layers having a commonbase-layer ID, respectively, “clusters”, the same secondinter-dependency syntax structure may be used to regulate theinter-dependencies within all clusters and between all clusters relatedto each other via the first inter-dependency syntax structure,separately. Alternatively, two instantiations of the secondinter-dependency syntax structure may be used to describe theinter-dependencies of the layers within the clusters on the one hand andbetween the layers of different clusters, on the other hand.Irrespective of the emphasis placed towards increased diversity orreduced side information overhead, the inter-dependency signalingconcept results in keeping the signaling overhead low.

One of the aspects of the present application is concerned with thesignaling of the layer ID which each of the packets of a multi-layeredvideo signal is associated with. In particular, this aspect achieves anefficient way of signaling this layer association, neverthelessmaintaining the backward compatibility with codecs according to which acertain value of the base layer-ID field is restricted to benon-extendable such as base layer-ID value 0 in the base layer-ID field.Instead of circumventing this restriction specifically with respect tothis non-extendable base layer-ID value, the layer-ID of portions of themulti-layer data stream is signaled in an extendable manner bysub-dividing the base layer-ID field into a first sub-field and a secondsub-field: whenever the first sub-field of the base layer-ID fieldfulfills a predetermined criterion, an extension layer-ID field isprovided, and if the first sub-field of the base layer-ID field does notfulfill the predetermined criterion, the extension layer-ID field isomitted. The aforementioned non-extendable base layer-ID value is“hidden” within the group of base layer-ID values for which the firstsub-field of the base layer-ID field does not fulfill the predeterminedcriterion, and accordingly this non-extendable base layer-ID value isnot handled separately, but rendered part of the former group. Rather,if the first sub-field of the base layer-ID field fulfills thepredetermined criterion, an extension value is derived from theextension layer-ID field signaled within the multi-layer data streamsuch that same lies within a first subset of a domain of extensionvalues, and if the first sub-field of the base layer-ID field does notfulfill the predetermined criterion, this extension value is set to avalue disjoint to the first subset of the domain of extension values.The layer which a respective portion is associated with is then indexedusing the extension value as well as the cluster value which is derivedfrom a second sub-field of the base layer-ID field. All in all, nosignaling efficiency has been lost despite the maintenance of thebackward compatibility.

One aspect of the present application is concerned with the signaling ofat least one characteristic for layers of a multi-layered video signalsuch as, for example, for each layer the indication of dependent layersto which the respective layer directly relates via inter-layerprediction, or the signaling of the afore-mentioned secondinter-dependency syntax structure. In accordance with this aspect, amaximum syntax element is signaled within the multi-layered video signalto indicate a maximally used value of an extension layer-ID field of thepackets of the multi-layered video signal, the scope of the maximumsyntax element being, for example, a predetermined portion of themulti-layered video signal extending, for example, across severalportions of the multi-layered video signal. Accordingly, according tothis aspect it is feasible for devices such as decoders or networkelements receiving the multi-layered video signal to gain, for arelatively large predetermined portion of the multi-layered videosignal, knowledge about the actually consumed portion of the possibledomain of possible values signalizable by the extension layer-ID fieldand the at least one characteristic does not need to be signaled foreach combination of base layer-ID field value and extension layer-IDfield value, but rather it is sufficient to signal the at least onecharacteristic for a maximum number of layers determined based on themaximum assumed value. Accordingly, the at least one characteristic doesnot need to be transmitted/signalized for layers with a layer ID, theextension layer-ID of which does not occur within the predeterminedportion of the multi-layered video signal. Beyond this, in accordancewith a further embodiment, the knowledge of the maximally assumed valuemay be used to reduce the side information overhead for signaling thelayer-ID of each portion, i.e. for reducing the bits needed to signalthe extension layer-ID field within the multi-layered video signal'spackets.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be detailed subsequentlyreferring to the appended drawings, in which:

FIG. 1 shows the syntax of an NAL unit header in HEVC;

FIG. 2 schematically illustrates an environment comprising amulti-layered video encoder, a network element and a multi-layered videodecoder, wherein a multi-layered video encoder, network element andmulti-layered video decoder may be improved by any of, or anycombination of, the concepts described in the following figures;

FIG. 3 schematically shows a network device and a concept of signalingin a cluster-wise manner layer interdependencies within themulti-layered data stream arriving at the network device;

FIG. 4 schematically illustrates the way of signaling the inter-layerdependencies in accordance with the concept of FIG. 3 ;

FIG. 5 schematically illustrates a more specific example to derive theinter-layer dependencies in case of using the signaling concept of FIG.3 ;

FIG. 6 shows a possible extension of the HEVC syntax of a slice segmentheader by showing a portion of the same extended by way of a syntaxelement indicating a layer-ID extension;

FIG. 7 shows an example for a VPS syntax exemplarily extended so as toimplement the signaling concept of FIG. 3 ;

FIGS. 8 to 10 show alternatives of the example of FIG. 7 ;

FIG. 11 schematically shows a device for setting up a further concept ofthe present application according to which a maximally assumed value ofan extension layer-ID field is signaled in the data stream;

FIG. 12 schematically illustrates a concept of FIG. 11 ;

FIG. 13 shows an example of a portion out of a slice segment header ofHEVC extended so as to include an extension layer-ID field

FIG. 14 shows an example of a VPS syntax extended so as to include asyntax element so as to indicate the length of the extension layer-IDfield of FIG. 13 ;

FIG. 15 shows an example of a VPS syntax extended so as to include asyntax element indicating the maximum value of the extension layer-IDfield;

FIGS. 16 a to 16 c show examples of portions out of syntax structures ofan SEI extension, SPS syntax or PPS syntax, extended so as to include anextension layer-ID field;

FIG. 17 shows an example of a VPS syntax signaling information on layerswithin the video data stream with cycling through syntax portionsthereof a number of times depending on the maximally assumed valueexplained with respect to FIGS. 11 and 12 ;

FIG. 18 shows a schematic exemplifying relationships between extensionlayer-ID field, base layer-ID field and layer-ID in accordance with aspecific example, in particular using layer_id_ext to denote theextension layer-ID field, nuh_layer_id so as to denote the base layer-IDfield, and layerID so as to denote the layer-ID, and the number ofrepresentable values of the extension layer-ID field being indicated bymax_layer_id_ext(plus1), here being exemplarily set to 3 by choosing thelength of the extension layer-ID field to be 2 by setting Length OfExtension equal to 2;

FIG. 19 shows an example of a portion out of a slice segment header,extended so as to comprise an extension layer-ID field;

FIG. 20 a shows an example of a VPS syntax extended so as to comprise asyntax element indicating a length of a subfield of the base layer-IDfield;

FIG. 20 b shows an example for a portion out a slice segment headersyntax, modified so as to conditionally explicitly signal an extensionlayer-ID field; and

FIG. 21 shows a flow diagram illustrating the functionality of a networkdevice deriving the layer-ID using the implemented relationship inaccordance with the concept of FIG. 18 .

DETAILED DESCRIPTION OF THE INVENTION

Before describing certain embodiments of the present application, anexemplary multi-layered video codec environment is described withrespect to FIG. 2 , in which the below-outlined embodiment and aspectsmay advantageously be used. Although this overview of an exemplarymulti-layered video codec environment is not to be understood asrestricting the embodiment outlined further below, the details andfunctionalities exemplarily provided with respect to the followingfigures shall be treated as describing a reservoir of possibleextensions of the embodiments further outlined below so as to result inmore detailed embodiments. In any case, the illustrative multi-layeredvideo codec environment presented hereinafter renders the understandingof advantages of the below-outlined embodiment easier.

FIG. 2 shows illustratively a multi-layered video encoder 200 whichreceives at its input a video material 202. For illustrative purposesonly, the video material 202 is illustrated as a multi-view video signalconveying video data for each of a plurality of views such as textureand depth for each picture of the video. Generally speaking, the videomaterial 202 may represent a multi-dimensional sampling of one scene.One dimension might be time t. That is, the video material 202temporally samples this scene, thereby resulting in one picture per timeinstant. Moreover, this scene may have been captured from differentviewpoints, thereby resulting in one video per view or in one pictureper view for each time instant. Further, the video material 202 mayprovide for each view, or each subset of the views, in addition to thespatial sampling of the texture of the scene, i.e. the spatial samplingof the colors of the scene, a depth map of the scene—for the respectiveview. FIG. 2 , for example, differentiates between depth maps, which areillustrated using hatching, and texture maps or images, which are drawnnon-hatched. However, it should be noted that none of the embodimentsoutlined further below are restricted to such multi-view materials.Rather, the illustration in FIG. 2 has only been provided for the sakeof alleviating the description brought forward below.

The multi-layered video encoder 200 is configured to encode the videomaterial 202 into a multi-layered data stream or video signal 204. Inparticular, multi-layered video encoder 200 codes the video materialinto the multi-layer data stream 204 at different layers, correspondingto different levels of information amount, using inter-layer prediction.This means the following. The multi-layered video encoder 200 generatesthe multi-layered data stream such that same comprises a plurality ofpackets 206, each of which is associated with one of different layers.The multi-layered video encoder 200 encodes, for example, into packets206 of a certain base layer some base level of information amount of thevideo material 202 such as, for example, merely the texture of one viewor the like. The association of the packets 206 to any of the variouslayers is, for example, derivable from a layer identification syntaxelement structure 208 within each packet 206, wherein the layeridentification syntax element structure 208 may, for example, becontained within a header 210 of the respective packet 206. For example,for illustration purposes, numbers 1, 2 and 3 are shown to be inscribedinto packets 206, thereby illustrating layers with layer-ID 1, 2 and 3,respectively. Among these layers, the layer with layer-ID 1 may, forexample, form the lowest or base layer providing the base level ofinformation amount. Packets 206 of other layers may increase theinformation amount available on the packets 206 of the base layer onlyin terms of one or more or various information types. For example,packets 206 of a certain layer may have a further view encoded thereinin addition to the views already coded within the base or referencelayer's packets 206. Alternatively, packets 206 of a certain layer mayhave depth information on a scene of the video material 202, such asdepth information of a view, the texture of which has already beenencoded into the packets of the base or—more generally—reference layerwhich may any lower layer. Likewise, surface reflectance information maybe coded into the packets of a further layer, i.e. a spatial sampling ofthe scene with respect to the surface reflectance of the objects withinthat scene irrespective of the illumination circumstances. And evenfurther, alpha blending information may be coded into the packets of afurther layer, i.e. a spatial sampling of the transparency of the scenepoints corresponding to the individual picture samples. Packets 206 of acertain layer may also add at a certain color component information ormay increase the spatial resolution, i.e. provide spatial resolutionrefinement. Similarly, packets 206 of a certain layer may simply providean SNR resolution refinement, i.e. increase the signal to noise ratio ofthe coded video material. In order to avoid redundancy within themulti-layered data stream 204 as far as the various layers areconcerned, inter-layer prediction is used by multi-layered video encoder200: that is, packets 206 of a certain layer have encoded therein aprediction residual with respect to an inter-layer prediction asobtained from one or more other layers, called reference layers withrespect to the former layer. The inter-layer prediction may be appliedto different entities such as syntax elements and parameters describingthe video material 202. For example, the prediction residual conveyed bypackets 206 of a certain layer may correct the inter-layer prediction asobtained from the one or more reference layers in the spatial domain ona per pixel basis. Alternatively, the multi-layered video encoder 200may, for example, use transform residual coding and the inter-layerprediction correction may take place in the transform domain, such asDCT domain on a per transform coefficient basis, for example.Alternatively or additionally, the multi-layered video encoder 200 maybe of a hybrid video codec type using spatial and/or temporal predictionand the inter-layer prediction may, for example, additionally oralternatively pertain to a refinement of motion vectors. Even further,the multi-layered video encoder 200 may use hierarchical multi-treesubdivision in order to subdivide the pictures of video material 202into coding blocks in units of which different prediction modes areapplied, such as spatial and temporal prediction, transform blocks inunits of which the aforementioned transform is performed on theprediction residual, and/or other blocks in units of which certaincoding parameters are set for coding the video material 202 and insteadof signaling subdivision information for a further layer anew, thesubdivisioning may either be adopted completely from any of the one ormore base layers, or a refinement with respect to that subdivisioninformation may be signaled.

By way of the inter-layer dependencies among the layers, the layers ofmulti-layer video signal 204 are interrelated with each other viabranches of a tree 212, the nodes of which are formed by theaforementioned layers. In order to convey information about theinter-layer dependencies among the layers, i.e. render information onthe inter-dependency tree 212 available at the decoding side, themulti-layered video signal 204 may have information thereon codedthereinto.

In FIG. 2 , for example, it is illustrated that multi-layered videoencoder 200 intersperses into the video data stream 204 high-levelsyntax packets 216 which comprise or convey information 214. Information214 may comprise an inter-dependency syntax structure so as to describethe inter-layer dependencies in tree 212.

However, the information on the tree 212 may, alternatively, be known orreconstructible by default at encoder and decoder. Accordingly,additionally or alternatively, the layer-related information 214 maycomprise information related to some layer-ID extension mechanism. Inparticular, while for many applications a moderate number of layers issufficient in order to form the multi-layered video signal 204, someother applications may suffer from a too small number of layerssignalizable by the layer identification syntax element structure 208.In other words, constructing the layer identification syntax elementstructure 208 such that same would also accommodate the huge number oflayers for those applications which involve such a high number oflayers, would disadvantageously result in a high side informationoverhead when considering the overall multiplicity of applications, andconstruing the layer identification syntax element 208 so that samemerely accommodates the moderate number of layers would exclude theother applications from being sufficiently supported by themulti-layered video codec underlying data stream 204. Accordingly, someextension mechanism may be used in order to be able to switch betweenmodes of the data stream 204, where the layer identification syntaxelement structure 208 accommodates merely the lower moderate number oflayers, and a mode where the layer identification syntax elementstructure even accommodates the high number of layers and information214 may additionally or alternatively participate in the switchingbetween these modes.

While the scope of validity of the high level syntax packet 216 mayencompass the whole multi-layered video signal 204, it is also possiblethat the predetermined portions to which packets 216 belong may besmaller, such as, for example, they may relate to chunks into which themulti-layered video signal 204 is temporally segmented, each chunk beinga sequence of pictures. For example, using DASH or another adaptedstreaming protocol, multi-layered video encoder 200 may change thenumber of layers coded into bitstream 204 in units of the just mentionedpredetermined portions, the inter-layer dependency in the form of treestructure 212 and/or the switching between extension or non-extensionwith respect to the layer identification syntax element structure 208.In any case, periodically transmitting the high level syntax packets 216enables recipients to, on a random access basis, be able to havemultiple random access time instances to start decoding multi-layeredvideo signal 214 inbetween.

FIG. 2 also shows exemplarily devices for receiving the multi-layereddata stream. An optional network element 218, for example, receives themulti-layered video data stream 204 and processes same in order toforward it to a recipient of the multi-layered video data stream 204.Accordingly, FIG. 2 also shows a multi-layered video decoder 220configured to decode the multi-layered video data stream 204. Bothentities, i.e. network element 218 and multi-layered video decoder 220form examples for devices for processing the multi-layered data stream204.

The network element 218 does not need to be able to decode themulti-layered data stream 204 using the inter-layer prediction betweenthe various layers thereof. Despite this, both network element 218 andmulti-layered video decoder 220, i.e. the recipient, need to be informedof the packets 206 association with the various layers and theinter-layer dependencies thereamong as defined by tree 21. Networkelement 218 discards, for example, packets 206 of data stream 204associated with layers where the additional information amount providedis, for example, preliminarily not needed within the further linkbetween network element 218 and multi-layered video decoder 220,respectively, for example, due to bitrate shortcomings within the link,the recipient's inability to reproduce the additional information amountor the like. Similarly, multi-layered video decoder 220 may also decideto discard some of the packets 206 of certain layers responsive toexternal parameters such as a currently available computation power, areproduction device's display performance such as spatial resolution,maximum number of input views or the like. That is, the device 218/220is able to read the layer identification syntax element structure 208 ofpackets 206 and, if present, to derive the inter-layer dependenciesamong the layers from information 214, and/or to switch between theextension mode or non-extension mode with respect to the layeridentification syntax element structure 208 responsive to theinformation 214 and/or to read other characteristics with respect to thelayers from information 214.

Beyond that, the multi-layered video decoder 220 is able to reconstructthe video material 202 from the inbound data stream 204 up to a certainlevel by collecting and using the information of packets 206 associatedwith that layer and all layers to which this layer is interrelated bythe inter-prediction dependencies (see the tree structure 212). That is,as outlined above multi-layered video decoder 220 may use packets 206 ofa certain layer as a prediction residual with respect to an inter-layerprediction which the multi-layered video decoder 220 derives from thepackets 206 of the one or more reference layers of that layer. In thisregard, multi-layered video decoder 220 may, as outlined above, be adecoder using transform residual decoding, hybrid video decoding,hierarchical multi-tree subdivisioning and/or other coding concepts.

For the sake of completeness only, FIG. 2 shows that multi-layered videoencoder 200 and data stream 204 may exemplarily be provided such thatpackets 206 are included within data stream 204 in such a manner thatpackets 206 belonging to different time instances or pictures are notinterleaved with each other so as to form, per time instant, consecutiveaccess units 222, each collecting the packets 206 belonging to therespective time instant of that access unit 222 so that packets 206belonging to any other time instant are within the data stream 204either before or subsequent to this access unit 222. However, it shouldbe noted that this restriction has been chosen merely for illustrationpurposes and a more relaxed arrangement of the packets 206 within datastream 204 may alternatively be chosen.

In the following, a possibility of signaling the inter-layerdependencies according to tree structure 212 is described in moredetail. As mentioned above, this signaling may be comprised by theinformation 214. In accordance with this possibility, the layers aregrouped into clusters. Within the data stream, the inter-layerdependencies among the clusters on the one hand and the layers withinthe clusters on the other hand are separately signaled. Insofar, thedescription brought forward below represents a possibility ofimplementing the information 214 of FIG. 2 in order to signal theinter-layer dependencies 212, but as already noted above, the detailsdescribed hereinafter with respect to FIG. 3 should not be restricted tothe details set out in FIG. 2 . Rather, FIG. 2 should be seen as apossible implementation reservoir for the description with respect toFIG. 3 .

FIG. 3 illustrates a network device 18, which may be the network element218 or the multi-layered video decoder 220 of FIG. 2 , and amulti-layered data stream 10 same is configured to process, such as datastream 204 of FIG. 2 .

FIG. 3 illustrates multi-layer data stream 10 as having coded thereintoa video material at different layers identified by layer-ID lID. Eachlayer corresponds to a different level of information amount.Possibilities in this regard have already been discussed with respect toFIG. 2 , but for the sake of an easier understanding one could think ofeach layer adding a certain view to data stream 10. However, this is notmeant to be restrictive for the description of FIG. 3 . Each layer may,alternatively, correspond to a combination of different measures for theinformation amount, such as, for example, a number of views and spatialresolution or the like.

The multi-layer data stream 10 is thus composed of a plurality ofpackets 12 which may, exemplarily, correspond to packets 206 of FIG. 2 .Such packets 12 (or 206) may be substreams allowing, for example,wavefront parallel processing of the individual pictures coded intovideo signal 10, which substreams may, in turn, be composed of smallerunits such as NAL units, slices or the like. However, it is submittedherewith that packets 12 or packets 206 of FIGS. 2 and 3 may also be NALunits, slices or other units.

Each packet 12 is associated with one of the different layers and inorder to reduce the bit consumption of the multi-layer data stream 10,inter-layer prediction is used so that each packet 12 merely adds a“residual” to packets of a certain subset of lower layers of the datastream 10. As shown at 14, the “lower layers” are, for each layer,illustrated by small dots.

In accordance with the embodiment shown in the FIG. 3 , the inter-layerprediction dependencies may be restricted as described in more detailbelow and illustrated in the figure using continuous lines 16. Inparticular, the layer-ID is formed by a concatenation of two values asdescribed in the following. In particular, the network device, whichreceives the multi-layer data stream 10, reads, per packet 12, alayer-ID syntax structure indicated using reference sign 208 in FIG. 2 ,but here composed of a base layer-ID field 20 and, on a conditionalbasis, —conditionally depending on the base layer-ID field or ahigh-level syntax element switching on and off extension mechanism, forexample—an extension layer-ID field 22. Exemplarily, merely packets 12having both fields are shown in the figure. But, as described above, oneor more values of the base layer-ID field 20 may signal the absence offield 22 for the respective packet 12. For example, the whole extensionfunctionality could be switchable on/off in the data stream via anextension flag in the data stream, and possible conditions fornecessitating or not necessitating the extension layer-ID field could bethat, as described above, the base layer-ID field is zero, is or has acertain bit set or not set, or assumes a certain value within asub-portion of field 20, or base layer-ID field having a value greaterthan, or smaller than some value. Further examples are set out below.

Based on the layer-ID syntax structure 20 and 22, the network device 18derives the layer-ID identifying the layer the respective packet 12 isassociated with, i.e. lID. Different possibilities will be describedbelow. The network device 18 also reads, from the multi-layered datastream 10, a first inter-dependency syntax structure containing, forexample, the above-identified direct_dependency_flags, and a secondinter-dependency syntax structure containing, for example, the belowdenoted direct_ext_dependency_flag and/orgeneral_direct_ext_dependency_flags. The first inter-dependency syntaxstructure indicates, in a binary manner, inter-dependencies betweenpairs of different values representable by the base layer-ID field 20,whereas the second inter-dependency syntax structure indicates, in thebinary manner, inter-dependencies between pairs of different valuesrepresentable by the extension layer-ID field. Based on bothinter-dependency syntax structures, the network device 18 then forms aninter-layer dependency matrix revealing inter-layer predictiondependencies between different layers, such as the one depicted at 14 inthe figure. The way the derivation may be done is illustrated in thefollowing using a pseudo code using for-next loops involving the firstand second inter-dependency syntax structures and is explained withrespect to the following figures, too. It should be noted, however, thatsignaled inter-layer prediction dependencies need not to be actuallyused in the data stream. Rather, the signaling of possible inter-layerprediction dependencies is for instructing the decoder or other networkdevices to take the steps resulting in that the interrelated packets areavailable in the appropriate order, i.e. the packets referred to byother packets according to the inter-layer prediction dependencies priorto the referring packets.

As will get clear from the description below, the construction of theinter-layer dependency matrix 14 may be performed such that the secondinter-dependency syntax structure is applied to all instances where theinter-layer dependency matrix 14 relates to inter-layer predictiondependencies between layers of layer-ID having been derived from, andthus being associated with, the same value within the base layer-IDfield 20. In accordance with another embodiment also described below inmore detail, the second inter-dependency syntax structure is read fromand is transmitted within the bitstream 10 several times such as, forexample, for each possible value of the base layer-ID field 20individually or for subsets of possible values of the base layer-IDfield 20, while associating the various instantiations of the secondinter-dependency syntax structure to the possible values of the baselayer-ID field 20 using indexing, for example, such as using includednuh_layer_id in the below presented example. In even other embodimentsthe structure is generalized in that all possible inter-layer predictiondependencies are allowed and describable by way of the secondinter-dependency syntax structure, namely by transmitting same per pairof different values of the base layer-ID field for which the firstinter-dependency syntax structure indicates inter-dependency existence.

In other words, and as further illustrated in FIG. 4 , FIG. 3 presentedan example for signaling the inter-layer dependencies according to whichseparate inter-dependency syntax structures 24 and 26 were used todescribe the inter-layer dependencies in an hierarchical manner with theinterface between the scope of both inter-dependency syntax structures24 and 26 coinciding with the interface between the scope of the baselayer-ID and extension layer-ID provided by fields 20 and 22 within eachpacket of the data stream. Base layer-ID and extension layer-ID asprovided by fields 20 and 22 uniquely define the layer-ID of therespective packet by which fields 20 and 22 are comprised. The set ofall representable layer-IDs representable by the combination ofextension layer-ID and base layer-ID are indicated within circle 28 bydots 30. That is, each dot 30 corresponds to a different couple of baselayer-ID and extension layer-ID. For example, the layer-ID may be theconcatenation of base-layer ID and extension layer-ID. Using the baselayer-ID as provided by field 20 only, the complete set 28 of layer-IDs30 are subdivided into disjoint sets 32 of layer-IDs called clusters 32in the following, wherein all layer-IDs belonging to a certain cluster32 have the same base layer-ID. As described previously with respect toFIG. 2 , the layers associated with layer-IDs 30 are linked to eachother in a tree-like manner due to inter-layer prediction with theseinter-layer dependencies being illustrated in FIG. 4 using dashed linesbetween dots 30. For the ease of understanding, only a subset of theactual inter-layer dependencies are illustrated in FIG. 4 .

In any case, the linking between two clusters 32 by way of aninter-layer dependency between a layer of the first cluster 32 and alayer of the second cluster 32 of this pair is indicated by way of thefirst inter-dependency syntax structure 24. In other words, the firstinter-dependency syntax structure 24 describes, coarsely orcluster-wise, the inter-dependencies between layers. In FIG. 4 , theseinter-dependencies are illustrated using continuous lines betweenclusters 32. All pairs of a first and a second cluster 32 wherein atleast one layer of the first cluster is linked to one cluster in thesecond cluster are interlinked and so indicated in the firstinter-dependency syntax structure 24. The second inter-dependency syntaxstructure 26 then clarifies which of the layers of pairs of clusters 32indicated as being interlinked by the first inter-dependency syntaxstructure 24, are actually related to each other by inter-layerprediction. That is, the second inter-dependency syntax structure 26clarifies the fine-granular inter-dependencies. The secondinter-dependency syntax structure 26, however, also defines theinter-dependencies among the layers within each cluster 32 individually,that is, the intra-cluster dependencies between layers. In FIG. 4 , forexample, six clusters 32 are shown, thereby resulting in 15 possiblepairs of clusters 32 for which the second inter-dependency syntaxstructure 26 regulates the fine-granular inter-dependencies betweenlayers of distinct clusters 32, plus six clusters for which the secondinter-dependency syntax structure 26 may regulate the inter-dependenciesinternally. As was outlined above and as will be further outlined below,there may accordingly be 1 to 21 instantiations 34 of the secondinter-dependency syntax structure at the maximum, namely one per clusterindividually and one per interlinked pair of clusters 32. Obviously, thesecond inter-dependency syntax structure 26 does not need to be signaledfor pairs of clusters 32 signaled to be non-linked by the firstinter-dependency syntax structure 24, thereby saving valuable sideinformation bitrate. It should be noted that in case of using the secondinter-dependency syntax structure 26 so as to describe the inter-layerdependencies of layers of different pairs of clusters 32, the numbers oflayers per cluster 32 should be equal for all clusters 32. If thelayer-ID is described using the concatenation of base layer-ID andextension layer-ID, this is the case. However, it should be mentionedthat theoretically the number of layers of the clusters 32 could varyamong the clusters 32. In that case, individual instantiations 34 of thesecond inter-dependency syntax structure 26 would, for example, besignaled within the data stream for each interlinked pair of clusters 32and at least one instantiation 34 would be transmitted for each clustersize.

FIG. 3 , for example, illustrated the case where the layer-ID wasobtained from base layer-ID and extension layer-ID by using the baselayer-ID as the most significant digit and the extension layer-ID as theless significant digit. FIG. 3 also illustrated the exemplary case whereone instantiation 34 was used to describe the inter-layer dependenciesof layers within one cluster and another instantiation 34 was used todescribe the dependencies between layers of different clusters. For thesake of completeness, it is noted that the matrix 14 of FIG. 3 has asmany lines and as many columns as layer-IDs. Merely the lower half belowthe diagonal is filled because any layer may merely be dependent on anyprevious, i.e. hierarchically lower, layer by way of inter-layerprediction. In the example of FIG. 3 , the column number corresponds tothe layer-ID of the layer which, by use of inter-layer prediction,depends on further layers, i.e. base layers, with these base layersindicated by, for example, binary ones, while binary zeroes indicatelayers not participating in inter-layer predicting the respective layer,the layer-ID of which corresponds to the current column. Insofar, in theembodiment of FIG. 3 , the second inter-dependency syntax structure 26more or less describes sub-matrices of matrix 14.

As was already described above, the first and second inter-dependencysyntax structures 24 and 26 may be comprised by information 214 withinhigh-level packets 216 (compare FIG. 2 ). FIG. 5 illustrates an example,where the first inter-dependency syntax structure 24 revealsinter-dependencies among the layer clusters as depicted at 36. Forexample, a cluster with base layer-ID 2 is dependent on clusters withbase layer-ID 2 and 1.

A first instantiation 34 of the second inter-dependency syntax structureis also present in the data stream and regulates the intra clusterdependencies among the layers depicted in FIG. 5 in form of a sub-matrix38. Further, in accordance with the example of FIG. 5 , the data streamalso comprises an instantiation 34 of the second inter-dependency syntaxstructure 26 which regulates the layer-wise inter-dependencies of layersof different clusters. In particular, the second instantiation maydescribe the dependencies between layers of different clusters via asub-matrix 40 having one row per enhancement layer-ID of the referencedcluster and one column per enhancement layer-ID of the referencingcluster.

In the example of FIG. 5 , sub-matrix 38 is placed at each positionwhere matrix 36 indicates an inter-dependency between clusters, i.e.where a 1 is positioned, and which clusters are of the same baselayer-ID, i.e. cluster lying on the diagonal of matrix 36, and thesub-matrix 40 is placed where matrix 36 indicates by a “1” aninter-dependency between clusters of different base layer-ID. The resultis shown at 42.

It should be noted that the description of the inter-layer dependenciesvia a matrix such as matrix 42 is merely one example for describing theinter-layer dependencies. Other descriptions may be used as well. Theway matrices 36 to 40 are coded by the first and second inter-dependencysyntax structures may by as follows: the first inter-dependency syntaxstructure 24 may signal a binary value for each coefficient of matrix 36beneath and including the diagonal. The Instantiation 34 of the secondinter-dependency syntax structure 26 indicating matrix 38, may signal abinary value for each coefficient of matrix 38 beneath and excluding thediagonal. The Instantiation 34 of the second inter-dependency syntaxstructure 26 indicating matrix 40 may signal a binary value for allcoefficients of matrix 40.

After having described possibilities as to how to signal the inter-layerdependencies, more detailed implementations are presented hereinafterexemplarily as an extension of the HEVC standard.

In particular, the layer identifier clustering and cluster dependencysignaling may be Bild into an existing codec as follows.

Two syntax elements nuh_layer_id and layer_id_ext may be used to groupthe coded layers within the video bitstream into so called clustersbased on properties such as spatial relation to each other, codingdependencies or others. The structuring of layer identifiers intocluster may allow for clusters with equally structured codingdependencies within the individual clusters, i.e. the codingdependencies within all or a subset of the defined cluster are the same.Signaling the dependencies within a cluster as an additional set ofdependency flags and combining them with existing dependency signaling(cp. direct_dependency_flag in the VPS extension of HEVC extensions) todetermine the reference layers of a particular layer may be done asdepicted in FIG. 6 .

direct_ext_dependency_flag[i][j] equal to 0 specifies that the layerwith extension index j is not a direct reference layer for the layerwith extension index i within the same layer cluster.direct_dependency_flag[i][j] equal to 1 specifies that the layer withextension index j may be a direct reference layer for the layer withextension index i within the same cluster. Whendirect_ext_dependency_flag[i][j] is not present for i and j in the rangeof 0 to (1<<layer_id_ext len)−1, it is inferred to be equal to 0.

An alternative syntax may limit the loop over thedirect_ext_depencency_flag syntax element with max_layer_id_ext to avoidsignaling coding dependencies to unused layer identifiers.

A more flexible signaling of the coding dependencies within and betweenclusters may be allowed as follows:

-   -   a) By signaling coding dependencies within clusters on a per        cluster basis.    -   b) By definition of a number of cluster dependency sets and        identification of the clusters (e.g. either by nuh_layer_id) to        which a defined cluster dependency set applies.    -   c) By hierarchical signaling of dependencies in a way that first        dependencies between clusters are signaled and second        dependencies of layers within two dependent clusters are        signaled.    -   d) By combination of b) and c). Therefore following data is        signaled:        -   1. Dependencies between clusters (e.g. pairs of reference            and dependent clusters).        -   2. A number of cluster dependency sets defining dependencies            between layers within a reference (referenced) cluster and a            dependent (referencing) cluster.        -   3. For each cluster dependency set an indication to which            cluster dependencies it applies (as signaled in 1).

Note that for c) and d) a cluster can depend on itself, when there aredependencies between layers within the cluster.

a) to d) allow to structure the space of layer identifiers when theproperty space used for structuring, e.g. spatial position of thecamera, is not uniformly used. One example is a two-dimensional cameraarray, which spatial density is not constant or which is notrectangular. The following gives syntax examples for signaling thedefinition of cluster dependency sets in multiple ways according to theabove.

a) FIG. 7 gives an exemplary embodiment in terms of video parameter setextension syntax that defines the coding dependencies within clusters ona per cluster basis.

direct_dependency_flag[i][j] equal to 0 specifies that the layer orcluster with index j is not a direct reference layer or referencecluster for the layer or cluster with index i.direct_dependency_flag[i][j] equal to 1 specifies that the layer orcluster with index j may be a direct reference layer or referencecluster for the layer or reference cluster with index i. Whendirect_dependency_flag[i][j] is not present for i and j in the range of0 to vps_max_layers_minus1, it is inferred to be equal to 0.

general_direct_ext_dependency_flag[i][j] equal to 0 specifies that thelayer with extension index j is not a direct reference layer for thelayer with extension index i within the same layer cluster.general_direct_dependency_flag[i][j] equal to 1 specifies that the layerwith extension index j may be a direct reference layer for the layerwith extension index i within the same cluster. Whengeneral_direct_ext_dependency_flag[i][j] is not present for i and j inthe range of 0 to max_layer_id_ext, it is inferred to be equal to 0.

direct_ext_dependency_flag[i][j][k] equal to 0 specifies that the layerwith extension index k is not a direct reference layer for the layerwith extension index j within the i-th layer cluster.direct_ext_dependency_flag[i][j][k] equal to 1 specifies that the layerwith extension index k may be a direct reference layer for the layerwith extension index j within the i-th layer cluster. Whendirect_ext_dependency_flag[i][j][k] is not present for i, j and k in therange of 0 to max_layer_id_ext, it is inferred to be equal to 0.

b) FIG. 8 gives an exemplary embodiment in terms of syntax in the videoparameter set extension that defines the coding dependencies withinclusters through signaling of a number of cluster dependency sets(number_of_cluster_dependency_sets), coding dependencies(direct_ext_dependency_flag) within each cluster dependency set and theapplying clusters (exemplary identified through a positive flag fortheir respective nuh_layer_id by the given syntax element includednuh_layer_id).

c) FIG. 9 gives an exemplary embodiment for hierarchical signaling ofdependencies in a way that first dependencies between clusters aresignaled and second only dependencies of layers within two dependentclusters are signaled.

direct_dependency_flag[i][j] equal to 0 specifies that the layer orcluster with index j is not a direct reference layer or referencecluster for the layer or cluster with index i.direct_dependency_flag[i][j] equal to 1 specifies that the layer orcluster with index j may be a direct reference layer or referencecluster for the layer or reference cluster with index i. Whendirect_dependency_flag[i][j] is not present for i and j in the range of0 to vps_max_layers_minus1, it is inferred to be equal to 0.

NOTE—If layer_id_ext len is greater than 0, direct_dependency_flag[i][j]signals dependencies between clusters, otherwise (layer_id_ext_len isequal to 0), direct_dependency_flag[i][j] signals dependencies forlayers.

direct_ext_dependency_flag[i][j][k][l] equal to 0 specifies that the1-th layer in the j-th cluster is not a direct reference layer for thek-th layer in the i-th cluster. direct_ext_dependency_flag[i][j][k][1]equal to 1 specifies that the 1-th layer in the j-th cluster may bedirect reference layer for the k-th layer in the i-th cluster. When notpresent direct_ext_dependency_flag[i][j][k][1] is inferred to be equalto 0.

d) FIG. 10 gives an exemplary embodiment for combination of b) and c).

direct_dependency_flag[i][j] equal to 0 specifies that the layer orcluster with index j is not a direct reference layer or referencecluster for the layer or cluster with index i.direct_dependency_flag[i][j] equal to 1 specifies that the layer orcluster with index j may be a direct reference layer or referencecluster for the layer or reference cluster with index i. Whendirect_dependency_flag[i][j] is not present for i and j in the range of0 to vps_max_layers_minus1, it is inferred to be equal to 0.

NOTE—If layer_id_ext len is greater than 0, direct_dependency_flag[i][j]signals dependencies between clusters, otherwise (layer_id_ext_len isequal to 0), direct_dependency_flag[i][j] signals dependencies forlayers.

direct_ext_dependency_flag[n][k][1] equal to 0 specifies that the 1-thlayer in the j-th cluster is not a direct reference layer for the k-thlayer in the i-th cluster when dependency_set_applies_flag[n][i][j] isequal to 1. direct_ext_dependency_flag[n][k][l] equal to 1 specifiesthat the 1-th layer in the j-th cluster may be a direct reference layerfor the k-th layer in the i-th cluster whendependency_set_applies_flag[n][i][j] is equal to 1.

dependency_set_applies_flag[n][i][j] equal to 0 specifies thatdependencies between layers in the i-th cluster and layers in the j-thcluster are not specified by direct_ext_dependency_flag[n][k][l].dependency_set_applies_flag[n][i][j] equal to 1 specifies thatdependencies between layers in the i-th cluster and layers in the j-thcluster are specified by direct_ext_dependency_flag[n][k][l]. When notpresent dependency_set_applies_flag[n][i][j] is inferred to be equal to0. When dependency_set_applies_flag[n][i][j] is equal to 0 for aparticular combination of i and j and all n in the range of 0 to(number_of_cluster_dependency_sets−1), inclusive, no layer in the j-thcluster is a direct reference layer of any layer in the i-th cluster.

Thus, in the above examples of implementing the cluster-basedinter-dependency signaling of FIGS. 2 to 5 into an HEVC extension, thepackets 206 of FIG. 2 were slices or slice segments and packets 216were, for example parameter sets such as video, picture or sequenceparameter sets. In accordance with the syntax example of FIG. 7 , thefirst inter-dependency syntax structure 24 is signaled using the syntaxelement direct_dependency_flag for each pair of clusters i, j, with i≤j,wherein vps_max_layers_minus1 is the maximum number of base layer-IDsminus 1. The syntax elements thus reveal the coarse matrix 36 of FIG. 5, for example. A syntax element, calledunique_cluster_dependencies_flag, 44 distinguishes between two options:either one instantiation of the second inter-dependency syntax structure26 is submitted and applied to all base layer-IDs, or one instantiationis transmitted for each base layer-ID i. In the first case, flagsgeneral_direct_ext_dependency_flag build up a sub-matrix according to 38for each base layer-ID commonly, and in the second case, flagsdirect_ext_dependency_flag build up a sub-matrix according to 38 foreach base layer-ID separately. unique_cluster_dependencies_flag switchesbetween both options. Accordingly, in the case of FIG. 7 , ifunique_cluster_dependencies_flag equals 0, the resulting sub-matrix 38is placed within matrix 36 at each position corresponding to equalreferencing and referenced cluster ID, where a 1 is indicated bydirect_dependency_flag. The remaining positions, where the firstinter-dependency syntax structure 24 indicate an interdependency betweenclusters of different cluster-IDs, may be filled using predeterminedsub-matrices, i.e. ones known by all participating devices such asencoder, network element and multi-layer video decoder, by default. Inthe case of unique_cluster_dependencies_flag being equal to 1, for eachposition where the first inter-dependency syntax structure 24 indicatesa 1 at positions of equal referenced and referencing cluster-ID,separate sub-matrix for this very cluster-ID is signaled and positionedat the respective position in matrix 42. Accordingly, max_layer_id_extcorresponds to the number of layers within each of thevps_max_layers_minus1 clusters.

In the case of FIG. 8 , the first inter-dependency syntax structure 24is transmitted in the same manner as in the example of FIG. 7 : for eachcluster-ID one flag per cluster ID being equal to or smaller than therespective cluster-ID. A flag cluster_dependency_sets_present_flag 46switches between the first option of FIG. 7 using syntax elementsgeneral_direct_ext_dependency_flag as already outlined above, and adifferent mode according to which the number of instantiations of thesecond inter-dependency syntax structure 26 submitted in the data streamusing syntax elements direct_ext_dependency_flag is explicitly signaledusing a syntax element called number-of-cluster-dependency-sets. Theseinstantiations are then associated with the cluster-IDs for which theseinstantiations shall apply, by explicitly listing the cluster-IDs forwhich each instantiation shall apply by using of syntax elementsincluded_nuh_layer_id 50. By this measure, the instantiations 34 of thesecond inter-dependency syntax structure 26 are not submitted for eachcluster-ID. Rather, merely a smaller number of instantiations of thesecond inter-dependency syntax structure 26 are submitted in case ofcluster_dependency_sets_present_flag being equal to 1, associating eachinstantiation to the cluster-ID which same is associated with. Again, asfar as inter-dependent pairs of different clusters are concerned,default rules may be applied for filling matrix 42 at such positions(below the cluster diagonal).

In accordance with the example of FIG. 9 , the signaling of the secondinter-dependency syntax structure is exemplarily signaled in aninterleaved manner with respect to the first inter-dependency syntaxstructure, but this interleaving may be left out in accordance with analternative embodiment. In any case, in the embodiment of FIG. 9 , asub-matrix 38 or 40, i.e. an instantiation 34 of the secondinter-dependency syntax structure 26, is submitted for each pair of(equal or unequal) cluster-IDs for which the first inter-dependencysyntax structure 24 indicates inter-cluster dependency. As is derivablefrom the syntax example, the transmission of sub-matrices 40 consumesmore bits direct_ext_dependency_flag than sub-matrices 38. This isderivable from the conditional clause 52 and conditional clause 54,respectively. In other words, in transmitting instantiations of thesecond inter-dependency syntax structure 26 for cluster-internalinter-dependency regulations, where the referencing cluster-ID indicatedby counter i equals the referenced cluster-ID indicated by j,direct_ext_dependency_flag is merely transmitted for combinations of theenhancement layer-ID for the reference and referencing layer for whichthe enhancement layer-ID of the referenced layer, here l, is smallerthan the enhancement layer-ID of the enhancement layer-ID of thereferenced layer, here k. That is, of sub-matrix 38,direct_ext_dependency_flag is merely transmitted for the positions belowthe diagonal of the sub-matrix 38. In the other case, i.e. forinstantiations of the second inter-dependency syntax structure 26 forinter-dependent cluster pairs i≠j, the complete sub-matrix istransmitted, i.e. a flag direct_ext_dependency_flag for each position ofsub-matrix 40.

The embodiment of FIG. 10 combines the special features of theembodiments of FIGS. 8 and 9 : complete sub-matrices, i.e.instantiations of the second inter-dependency syntax structure 26 aresubmitted, namely number_of_cluster_dependency_sets in number, and theneach of these instantiations are associated with the grid sites ofmatrix 36, where the respective instantiation of the second syntaxstructure 26 shall apply (among those for which the first syntaxstructure 24 indicates the presence of cluster-independency (compare 56)by a one, for example.

In even other words, as described with respect to FIG. 9 , a pluralityof instantiations 34 of the second inter-dependency structure 26 may beread from the multi-layer data stream, namely for pairs (j,k) of values0<k≤j≤n<N, with N being the number of values representable by the baselayer-ID. n may set to be equal to N, but as also described herein, anexplicit signaling of the actually used cluster cardinality may be usedso as to restrict the transmission overhead. The pairs (j,k) aretraversed (see for loops over i and j at 24 in FIG. 9 ) and reading thesecond inter-dependency structure for is performed or suppresses forpair (j,k) conditionally depending on the first inter-dependency syntaxstructure indicating a presence or non-presence of interdependencybetween this pair (see “if direct_dependency_flag[ij]” in FIG. 9 , wherei,j correspond to j and k, respectively), wherein the instantiation 34of the second inter-dependency structure 26 read for pairs (j,k) withj=k, indicates which layers among those having base layer-ID j areinter-layer prediction dependent on other layers among those having baselayer-ID j, and the instantiation 34 of the second inter-dependencystructure 26 read for a pair of different values (j,k) with j>k,indicates which layers among those having base layer-ID j areinter-layer prediction dependent on layers among those having baselayer-ID k. Similar to FIG. 7 , however, at may also be that oneinstantiation of the second inter-dependency structure 26 is read fromthe multi-layer data stream, the one instantiation of the secondinter-dependency structure 26 indicating commonly for each cluster valuepair (j, j), which layers among those having base layer-ID j areinter-layer prediction dependent on other layers among those having baselayer-ID j, as it is the case in FIG. 7 ifunique_cluster_dependencies_flag equals 0, or commonly for pairs (j,k)with j unequal k, for which the first inter-dependency syntax structure(24) indicates a presence of interdependency between the respectivepair, which layers among those having base layer-ID j are inter-layerprediction dependent on layers among those having base layer-ID k,respectively. Denoting the number of values representable by theextension layer-ID 22 as M, the second inter-dependency syntax structuremay comprises one binary value for each pair (p,q) of extension values0<q<p≤m<M, in order to indicate which layers among those having baselayer-ID i are inter-layer prediction dependent on other layers amongthose having base layer-ID i, and one binary value for each pair ofextension values 0<p,q≤m<M, in order to indicate which layer withextension value p and having base layer-ID j is inter-layer predictiondependent on layer with extension value q and having base layer-ID k.Again, as described elsewhere in the specification, m may set to beequal to M, but as also described herein, an explicit signaling of theactually used extension cardinality may be used so as to restrict thetransmission overhead. Further, as shown in FIGS. 8 and 10 , an indexsyntax structure 50 may be used so as to associate instantiations of thesecond inter-dependency structure to the individual pairs (j,k) of baselayer-ID values. Here, FIG. 10 reveals that the association syntaxstructure may comprise a flag for each pair (j,k) and the parsing of theassociation syntax structure may involve skipping pairs (j,k) of values0<k≤j≤n<N for which the first inter-dependency syntax structureindicates the non-presence of interdependency (independency) betweenpair (j,k).

That is, what has been described in the above section could be denotedas a “layer identifier clustering and cluster dependency signaling”. Inthis regard, the device may be a video decoder. The video decoder, thusoperating, would be able to select packets of the bitstream identifiedby (the extended) layerID for decoding. The device could, however,alternatively be a network element which would be able to discardpackets of a bitstream that are identified by (the extended) layerIDbased on external factors such as network utilization, knowledge aboutdecoders, etc.

Naturally, above outlined concept would also impact an existing encoderof a codec to be improved by the concept: The encoder such as the one ofFIG. 2 , would be configured to encode into a multi-layered data stream10 a video material at different layers, corresponding to differentlevels of information amount, using inter-layer prediction, themulti-layer data stream comprising a plurality of packets 12, each ofwhich is associated with one of the different layers, each layer beingindexed by a base layer ID 20 or a base layer ID 20 and an extensionlayer ID 22, and to insert, into the multi-layered data stream, thefirst inter-dependency syntax structure 24 indicating interdependenciesbetween pairs of different values representable by the base layer-ID 20,and a second inter-dependency syntax structure 26 indicatinginter-dependencies between pairs of different values representable bythe extension layer-ID 22, with setting the first and secondinter-dependency syntax structures such that, based on the first andsecond inter-dependency syntax structures, an inter-layer dependencydescription 14 revealing possible inter-layer prediction dependenciesbetween the different layers is constructible.

The details described next concern a possibility to signal layer-relatedinformation in an efficient manner, wherein such layer-relatedinformation may, for example, include information concerning inter-layerdependencies as outlined with respect to FIGS. 2 to 10 . As a startingpoint, the situation often presented above is used, according to whichthe layer identification syntax element structure 208 comprised by eachpacket 206 (compare FIG. 2 ) is composed of a base layer-ID field 108and an extension layer-ID field 112 as depicted in FIG. 11 . FIG. 11shows a device 100 for processing a multi-layered data stream such as anetwork element 218 or a multi-layered video decoder 220 as shown inFIG. 2 . The device 100 is shown as comprising a receiver 102 configuredto receive a multi-layered video signal composed of a sequence ofpackets, each of which comprises a layer identification syntax element.

The multi-layered video signal 104 (corresponding to 204 in FIG. 2 )received by receiver 102 is composed of a sequence of packets 106(corresponding to 206 in FIG. 2 ), each of which comprises a layeridentification syntax element structure (corresponding to 208 in FIG. 2) comprising at least a base layer-ID field 108 and, unconditionally orconditionally depending on the value of a high-level syntax element orthe base layer-ID field 108, an extension layer-ID field 112.

The packets 106 may, for example, be NAL units, slices, tiles, picturesor other packets into which the multi-layered video signal 104 issubdivided. A layer identification extender 110 also comprised by device100 reads, for a predetermined portion of the multi-layered video signalcomprising a plurality of packets 106, such as portion 113, a maximumsyntax element from the multi-layered video signal, indicating amaximally assumed value of the extension layer-ID field 112 of thepackets 108 comprised by the predetermined portion 113 of themulti-layered video signal. The predetermined portion 113 may, forexample, be a coded video sequence, a chunk, a group of pictures or thelike. The maximum syntax element 114 may be contained within a specialpacket of portion 113 such as, for example, a VPS NAL unit. For each ofthe packets within the predetermined portion 113, the extender 110determines a layer ID for the respective packet based on the layer-IDsyntax structure composed of 108 and, conditionally, 112. For example, aconcatenation of both values of both syntax elements 108 and 112 mayresult in the layer ID.

The extender 110 may derive a bit length of the extension layer-ID fieldusing the maximum syntax element as is illustrated above. Alternatively,an extra syntax element may be used to this end. Even alternatively, thenumber of bits for representing the extension layer-ID field 112 may befixed by default.

Extender 110 then determines the maximum number of layers within theportion 112 based on the maximum assumed value. For example, extender110 also reads a syntax element from the data stream 104 indicating forportion 113 the maximum assumed value for the base layer-ID field 108 ofthe packets 106 within portion 113 and combines both maximum assumedvalues for determining the maximum number of layers.

The extender 110 determines at least one characteristic for each of themaximum number of layers by iteratively parsing a layer characterizingsyntax portion of the multi-layered video signal a number of times equalto the maximum number of layers. Advantageously, the data stream 104does not need to signal the at least one characteristic for eachpossible value, i.e. not for the whole cardinality of the extensionlayer-ID field 112, but merely for the actually used sub-portion out ofthis cardinality. Additionally, a maximum syntax element 114 may even beused so as to derive of the number of representation bits of theextension layer-ID field 112 within the data stream 104 asjust-described. The “characteristic” may, for example, be theinter-layer prediction to other layers or the like.

In other words, with respect to FIG. 11 it has been described that it ispossible to transmit within the data stream 204/104 an explicitsignalization as to how many out of the representable states of theextension layer-ID field are actually assumed when considering allpackets 106/206 within the predetermined portion 113 of interest. Forexample, let the extension layer-ID field 112 be a field of N bits, then2^(N) extension layer-IDs 120 would be representable by the extensionlayer-ID field 112. The length of field 112 may be fixed or may besignaled and set within the data stream by way of a certain high-levelsyntax element indicated using dashed lines 122 in FIG. 12 . For somereason, however, not all of the available extension layer-IDs may havebeen used within the predetermined portion 113 of interest. In FIG. 12 ,for example, it is illustrated that merely 7 representable extensionlayer-IDs are used, namely by way of hatching.

In accordance with FIG. 11 , a maximum syntax element 114 indicates thisnumber of actually used/assumed values of the extension layer-ID field112. Owing to this syntax element 114, it is feasible to insertlayer-specific information merely with respect to the actually assumednumber of extension layer-IDs into the data stream rather than for thecomplete number of representable values of extension layer-ID field 112.Thus, as illustrated at 122 in FIG. 12 , a device parsing themulti-layered video signal may, for example, parse a certain layercharacterizing syntax portion 124 of the multi-layered video signal,merely a number of times equal to the maximum number of layers asindicated by syntax element 114 rather than the complete numbers ofactually available/representable values of the extension layer-ID field.The side information amount consumed with the data stream may,accordingly, kept lower. For example, in the examples of FIGS. 6 to 10 ,it would be feasible to set max_layer_id_ext to be equal to themaximally assumed number of extension states as indicated by syntaxelement 114 rather than setting this value to be equal to the maximumnumber of representable values of the extension layer-ID field 112.Thus, the “characterizing syntax portion” 124 may be embodied by thepreviously presented flag direct_ext_dependency_flag.

As already outlined above, the existence/signaling of the bit length offield 112 is optional. In any case, the granularity/fidelity at whichsyntax element 114 indicates the number of actually used/activeextension layer-IDs is increased compared to the coarser setting of thenumber of available/representable values of the extension layer-ID field112 as set by syntax element 122. Advantageously the fidelity orgranularity at which syntax element 114 indicates the number of actuallyassumed values of the extension layer-ID field 112 is fine enough inorder to indicate the exact number of actually assumed values.Alternatively, a fidelity/granularity somewhere between one and thefidelity/granularity of the signalization of the maximum number ofavailable/representable values of field 112 as provided by syntaxelement 122 would be feasible as well. In other words, the maximumsyntax element 114 may indicates the maximally assumed value of theextension layer-ID field 108 in units smaller than (n−1)² with n beingthe bit length of the extension layer-ID field, or may even be one.

Thus, in accordance with FIGS. 11 and 12 it has been described that adevice, such as a network element or a multi-layered video decoder, maycomprise a receiver 102 and an extender 110 wherein the receiver 102 isconfigured to receive a multi-layered video signal 104 composed of asequence of packets 106 each of which comprises a layer identificationsyntax element structure composed of a base layer-ID field and, possiblymerely on a conditional basis, an extension layer-ID field 112, whereinthe layer identification extender 110 is configured to read, for apredetermined portion 113 of the multi-layered video signal comprising aplurality of packets 106 out of the sequence of packets 106, a maximumsyntax element 114 from the multi-layered video signal 104, the maximumsyntax element 114 indicating a maximally assumed value of the extensionlayer-ID field 112 among the packets 106 comprised by the predeterminedportion 113. The extender 110 determines, for each of the packets 106within the predetermined portion 113, the layer-ID for the respectivepacket 106 based on the layer identification syntax element structuresuch as by concatenating base layer-ID and extension layer-ID of field108 and 112 as outlined above. The extender 110 also determines amaximum number of layers within the predetermined portion 113 based onthe maximally assumed value and determines at least one characteristicfor each of the maximum number of layers by iteratively parsing a layercharacterizing syntax portion 104 a number of times equal to the maximumnumber of layers. The “maximum number of layers” may denote the maximumnumber of layers per cluster when combining the embodiment of FIGS. 11and 12 with the embodiment outlined above with respect to FIGS. 2 to 10, in which case the determination of the “maximum number of layers” isfor example directly adopting the maximally assumed value as the“maximum number of layers”, and the “at least one characteristic” may bethe number of flags indicating the inter-layer dependencies within thesecond inter-dependency syntax structure. Another example of acharacteristic, however, may also be a signalization within the datastream of corresponding coding parameters set for the individuallayer-IDs. In that case, the “maximum number of layers” may bedetermined to be equal to the number of used or representable baselayer-IDs times the maximally assumed value, i.e. the number of actuallyused/assumed extension layer-IDs. Other examples are feasible as well.In any case, the transmission of the actually assumed value by way ofsyntax element 114 enables to save valuable side information bit rate.

The concept of FIGS. 11 and 12 may, as already outlined above, becombined with the concept outlined above with respect to FIGS. 2 to 10or may be used isolatedly without using the concept previouslydescribed.

Similarly to the above description brought forward with respect to FIGS.2 to 10 , however, the following description reveals a possibility howto build the concept of FIGS. 11 and 12 into an existing video codecsuch as an extension of HEVC.

In HEVC, a layer identifier variable, referred to as LayerID, may bederived from the explicitly signaled layer identifier in the header ofvideo and meta data packets, e.g. nuh_layer_id and additionalinformation signaled in the each chunk of video or meta data packets,forming a new variable LayerId of the specific data packet to be usedfor its identification.

The additional information signaled according to FIGS. 11 and 12 mayinvolve an additional syntax element, cp. layer_id_ext, e.g. within somepart of the header or payload of the video or meta data packet, giventhat the specific video or meta data packet belongs to an enhancementlayer, thereby keeping compatibility with base layer only legacydevices. A default value can be assumed for layer_id_ext when the videoor meta data packet belongs to the base layer but is processed by anenhancement layer processing capable device.

FIG. 13 gives an example for a slice-segment header that signals theadditional information as the syntax element layer_id_ext in the sliceheader that is contained in a video data packet of enhancement layer(c.p. nuh_layer_id>0) of a HEVC coded video.

A mathematical operation is used in order to combine nuh_layer_id andlayer_id_ext into LayerId that allows a larger identifier space and isunique for a unique combination of two values of the nuh_layer_id andlayer_id_ext. The mathematical operation can for example be using thebits of nuh_layer_id as MSB and the layer_id_ext as LSB of the LayerIdvariable or vice versa, e.g. as following.

The variable LayerId is determined as follows, where LengthOfExtensionis the length of the layer_id_ext syntax elements in terms of bits.LayerId=(nuh_layer_id<<LengthOfExtension)+layer_id_ext

The concept of FIGS. 11 and 12 may include explicit signaling of theamount of bits that is used to transmit the additional information (cpLengthOfExtension above) in some part of the header or payload of thevideo or meta data packet in order to make efficient use of thetransmitted amount of data. FIG. 14 gives exemplary syntax and semanticsof the LengthOfExtension value signaling in meta data packets such asthe HEVC Video Parameter Set extension syntax.

layer_id_ext_len indicates the number of bits used for extending theLayerId range.

The concept of FIGS. 11 and 12 includes explicit signaling of themaximum value of the layer_id_ext syntax element used in the coded videobitstream, e.g. max_layer_id_ext. Based thereon, the lowest possibleamount of bits that may be used for transmitting the additionalinformation (cp LengthOfExtension above) in some part of the header orpayload of the video or meta data packet may be derived in order to makeefficient use of the transmitted amount of data. FIG. 15 gives exemplarysyntax and semantics of the maximum value of the layer_id_ext syntaxelement value signaling in meta data packets such as the HEVC VideoParameter Set extension syntax.

max_layer_id_ext indicates the maximum value of layer_id_ext syntaxelement in any slice header within the coded video sequence.

Two exemplary variants of the semantics of the layer_id_ext syntaxelement depending on the above extensions of the invention are given inthe following.

layer_id_ext is used in combination with nuh_layer_id to identify thelayer. The syntax element layer_id_ext is coded using layer_id_ext_lenbits. When not present, its value is inferred to 0.

layer_id_ext is used in combination with nuh_layer_id to identify thelayer. The syntax element layer_id_ext is coded Ceil(Log2(max_layer_id_ext)) bits. When not present, its value is inferred to 0

Layer identifier of parameter set or SEI messages of the HEVC videocodec standard can be extended given in FIG. 16 a-16 c without parsingdependencies.

In a layered coding scenario, signaling that is related to the layeridentifier or the maximum number thereof, is adjusted to cover theextended layer identifier space (cp. value range of LayerId) instead ofthe regular explicit layer identifier space (cp. value range ofnuh_layer_id), as exemplarily given with the syntax table for the videoparameter set extension in FIG. 17 , where the variable VpsMaxNumLayersis exemplarily given as follows.

The variable VpsMaxNumLayers specifying the maximum number of layer inthe extended layer range is set equal to(vps_max_layers_minus1)*2^(layer_id_ext_len).+1 or

The variable VpsMaxNumLayers specifying the maximum number of layer inthe extended layer range is set equal to(vps_max_layers_minus1)*max_layer_id_ext.+1

layer_id_in_nalu[i] specifies the value of the LayerId value associatedwith VCL NAL units of the i-th layer. For i in a range from 0 toVpsMaxNumLayers−1, inclusive, when not present, the value oflayer_id_in_nalu[i] is inferred to be equal to i.

When i is greater than 0, layer_id_in_nalu[i] shall be greater thanlayer_id_in_nalu[i−1].

For i in a range from 0 to vps_max_layers_minus1, inclusive, thevariable LayerIdInVps[layer_id_in_nalu[i]] is set equal to i.

dimension_id[i][j] specifies the identifier of the j-th presentscalability dimension type of the i-th layer. The number of bits usedfor the representation of dimension id[i][j] is dimension_id_len_minus1[j]+1 bits. When dimension_id[i][j] is not present for j in the range of0 to NumScalabilityTypes−1, inclusive, dimension_id[i][j] is inferred tobe equal to ((layer_id_in_nalu[i] &((1<<dimBitOffset[j+1])−1))>>dimBitOffset[j]).

The variable Scalabilityld[i][smIdx] specifying the identifier of thesmIdx-th scalability dimension type of the i-th layer, the variableViewld[layer_id_in_nalu[i]] specifying the view identifier of the i-thlayer and the variable ViewScalExtLayerFlag specifying whether the i-thlayer is a view scalability extension layer are derived as follows:

for ( i = 1; i <= VpsMaxNumLayers; i++) {  lId = layer_id_in_nalu[ i ] for( smIdx= 0, j =0; smIdx < 16; smIdx ++ )   if( ( i ! = 0 ) &&scalability_mask[ smIdx ] )    ScalabilityId[ i ][ smIdx ] =dimension_id[ i ][ j++ ]   else    ScalabilityId[ i ][ smIdx ] = 0 ViewId[ lId ] = ScalabilityId[ i ][ 0 ]  ViewScalExtLayerFlag[ lId ] =( ViewId[ lId ] != ViewId[ 0 ] ) }

Naturally, above outlined concept would also impact an existing encoderof a codec to be improved by the concept: The encoder such as the one ofFIG. 2 , would be configured to encode the video signal into themulti-layered video signal 104 composed of a sequence of packets 106each of which comprises a layer identification syntax element structurecomposed of a base layer-ID field 108 and an extension layer-ID field112, and to insert, for a predetermined portion 113 of the multi-layeredvideo signal comprising a plurality of packets, the maximum syntaxelement 114 into the multi-layered video signal, indicating themaximally assumed value of the extension layer-ID field 108 of thepackets comprised by the predetermined portion 113 of the multi-layeredvideo signal, determine a maximum number of layers within thepredetermined portion of the multi-layered video signal based on themaximally assumed value, and signal at least one characteristic for eachof the maximum number of layers in the multi-layered video signal 104 byiteratively writing a layer characterizing syntax portion 124 of themulti-layered video signal a number of times equal to the maximum numberof layers. The encoder would, for example, delay the output of thestream 104 till having finished the coding of portion 113.

The following figures deal with a concept of signaling the layer ID ofthe packets contained in the multi-layer data stream and a possibilityof achieving an efficient of signaling this layer-ID using the extensionmechanism without compromising backward compatibility in the case of anecessity which, due to any reasoning, requests that a certain value ofthe base layer ID, such as 0, is excluded from the possibility of havingappended thereto an extension layer-ID. The latter necessity results inthe following asymmetry: for almost all possible values of the baselayer-ID, appending an extension layer-ID is feasible. This results inthe following asymmetry: for almost all values of the base layer-ID, theappending of an extension layer-ID is feasible, thereby resulting in thenumber of these almost all values of base layer-ID times the number ofrepresentable values of the extension layer-ID representable layer-IDvalues. The restricted base layer-ID value, however, excludes thepossibility of appending any extension layer-ID. Accordingly, eitherthis asymmetry is accepted, or the forbidden value of the base layer-IDis not used in case of using the extension mechanism. The solution setout below overcomes this problem in a more efficient way.

This time, the description of this next concept is immediately outlinedwith respect to a possible extension of the HEVC standard where the justoutlined restriction exactly applies. However, the concept outlinedbelow is also transferrable to other codecs which, for example, imposesimilar restrictions onto any layer indication extension mechanism.

A restriction of HEVC is, as outlined above, that the base layer cannotbe extended. Hence to comply with the HEVC specification the layer idextension value layer_id_ext cannot be signaled for layers withnuh_layer_id equal to 0 and therefore the cluster associated withnuh_layer_id equal to 0 can only include the base layer itself. Such anunequal number of layers in different clusters might be undesirable.

The issue can be resolved by the below outlined concept, which derivesclusters and layer id extension values within the clusters in adifferent way than described above. Since it is not possible to havemultiple layers for nuh_layer_id equal to 0, the restriction to not havemultiple layers is extended to other nuh_layer_id values not equal to 0.Layers belonging to these restricted nuh_layer_id values are thencombined with layers from unrestricted nuh_layer_id values to formclusters with an equal number of layers.

The basic idea is that nuh_layer_id is split into two values by, forexample, bit masking. One value is used to determine the cluster. Thisway different nuh_layer_id values can belong to the same cluster. Theother value is utilized to determine the presence of the layer_id_extvalue (hence to determine if multiple layers can belong to thenuh_layer_id value) and to determine an additional offset to thelayer_id_ext value in LayerId derivation.

An example for an embodiment of the basic idea is given in the followingand illustrated in FIG. 18 . FIG. 18 , thus, provides an example for therelationships between layer_id_ext, nuh_layer_id and LayerId for theexemplary case of max_layer_id_ext equal to 3 (LengthOfExtension equalto 2).

In the example only, the five most significant bits of the six bitnuh_layer_id value are used to determine the cluster, hence twonuh_layer_id values with different least significant bits belong to onecluster.

If the least significant bit is equal to 0, no layer_id_ext values aresignaled for the nuh_layer_id and only one layer belongs to thenuh_layer_id. By definition this layer is the layer with layer_id_extequal to 0 within the cluster specified by the five most significantbits.

Otherwise, if the least significant bit is equal to 1, multiple layersbelong to the nuh_layer_id value. For each layer a layer_id_ext value issignaled explicitly and each layer belongs to the cluster specified bythe five most significant bits.

The syntax table of FIG. 19 might be used for the embodiment. It shows aheader of a packet 206, i.e. a slice segment header—or, to be moreprecise, a fragment thereof.

Layer_id_ext is not present when the least significant bit ofnuh_layer_id is equal to 0. When it is not present, it is inferred to beequal to 0. The variable LayerId is determined as follows, whereLengthOfExtension is the length of the layer_id_ext syntax elements interms of bits.LayerId=((nuh_layer_id>>1)<<LengthOfExtension)+layer_id_ext

In another embodiment a variable add_layer_id_ext_len is signaled in thebitstream, e.g. in the VPS. (Please note that the previously describedembodiment is a special case of this embodiment withadd_layer_id_ext_len equal to 1). The (6−add_layer_id_ext_len) mostsignificant bits of the six bit nuh_layer_id syntax element are useddetermine the cluster. The least add_layer_id_ext_len significant bitsspecify whether layer_id_ext is inferred. Furthermore they specify anoffset additional to the layer_id_ext value. The syntax tables of FIGS.20 a and 20 b are used to demonstrate the embodiment:

In the embodiment layer_id_ext is only present when theadd_layer_id_ext_len least significant bits of nuh_layer_id are equalto 1. When layer_id_ext is not present it is inferred to be equal to 0.The variable LayerId is determined as follows, where LengthOfExtensionis the length of the layer_id_ext syntax elements in terms of bits.LayerId=((nuh_layer_id>>add_layer_id_ext_len)<<LengthOfExtension)+layer_id_ext+layerIdExtOffset

Thus, in accordance with the concept outlined above with respect toFIGS. 18 to 20 b a device, such as a network element of multi-layeredvideo decoder such as those depicted in FIG. 2 , configured to process amulti-layer data stream into which a video material is coded atdifferent layers, corresponding to different levels of informationamount, using inter-layer prediction, would do the following for each ofthe packets comprised by the multi-layer data stream, each packet beingassociated with one of different layers, as already described above.

In particular, as shown in FIG. 21 , the device reads the base layer-IDfield 308 from the multi-layer data stream or, to be more precise, fromthe current packet. The base layer-ID field comprises a first sub-field308 a and a second sub-field 308 b. For example, the first sub-field maybe formed by one or more least significant bits of field 308, while thesecond sub-field 308 b is formed by the other most significant bits offield 308. As indicated in the example of FIG. 20 b , a higher-levelsyntax element, called add_layer_id_ext_len 310 may define the length ofsub-field 308 a.

After reading the base layer-ID field 308 in step 312, the device checkswhether the sub-field 308 a fulfills a predetermined criterion in step314. For example, the criterion is fulfilled if the sub-field is one oris an “all-ones sequences”. In case of the predetermined criterion beingfulfilled, the device reads an extension layer-ID field 316 from thedata stream or, to be more precise, from the current packet, in step 318and derives in step 320 an extension value using the extension layer-IDfield so that the extension value lies within a first subset of a domainof extension values. The derivation in step 320 may correspond to a merecopying of the bit representation within field 316. It may be arequirement of the data stream that the value within extension layer-IDfield 316 does not assume a certain value such as 0. It is the lattervalue which is reserved for the case that the base layer-ID field's 308sub-field 308 a does not fulfill the criterion as will be outlinedfurther below.

After step 320, the device derives a cluster value from the secondsubfield 308 b by mapping from a domain of possible values of the secondsub-field 308 b onto a domain of cluster values. The layer which thecurrent packet is associated with, i.e. the one comprising fields 308and 316 is then indexed using the cluster and the extension valuederived in steps 320 and 322 in step 324. Cluster and extension valuesare, for example, simply concatenated to each other using, for example,the cluster value for the most significant digit. The derivation of thecluster value 322 on the basis of sub-field 308 b may, for example, alsoconsist of a direct usage of the bits of sub-field 308 b as a binaryrepresentation of the cluster value.

However, if the check in step 314 results in the predetermined criterionnot being met, the device refrains from reading an extension layer-IDfield from the multi-layer data stream for the current packet. Rather,the device sets the extension value onto a value disjoint to the firstsubset of the domain of the extension values in step 326, such as bysetting the extension value to zero, and derives the cluster value fromthe second sub-field 308 b by mapping from the domain of possible valuesof the second sub-field 308 b onto the domain of cluster values. Thederivations in step 322 and 328 may be performed using the same mapping.Then, the layer-ID of the current packet is determined from the clusterand extension values as set and derived in steps 326 and 328, whereinthe determination of the layer-ID may again be performed in the samemanner as if the cluster and extension values were derived in steps 320and 322.

Relating the description of FIG. 21 to the embodiment of FIG. 18 , thefirst sub-field 308 a may thus be 1 bit long and in particular be theLSB of field 308. The predetermined criterion may be fulfilled if thisLSB 308 a is won and the aforementioned domain of extension valuesspanned by extension layer-ID field 316 would be {0, 1, 2, 3} as, incase of FIG. 18 , layer_id_ext, i.e. field 316, is 2 bits long.Accordingly, the first subset of the domain of extension values, i.e.the codomain of the derivation in step 320, would be {1, 2, 3}. Inparticular, the derivation in step 320 is, in the example of FIG. 18 ,embodied by way of a unitary matrix applied to the extension layer-IDfield. Alternatively, some other computation may be used, such as addingthe value of the first sub-field 308 a to the result of the unitarymatrix application. Accordingly, in step 326 the extension value is setto a value disjoint to {1, 2, 3}, namely 0. The mapping used in thederivations 322 and 328 may be the same, but may be differentalternatively. The mapping may in particular be bijective.

The determination of the index in 324 may use the cluster value to set amore significant digit and the extension value to set a lowersignificant digit as illustrated in FIG. 18 . Thus, as illustrated inFIG. 18 , the layer_id_ext field, i.e. the extension layer-ID field 316,is merely present in case of the LSB of nuh_layer_id, i.e. base layer-IDfield 308, being 1. Otherwise, the layer_id_ext field is inferred to bezero. Accordingly, the mapping of layer-IDs to the combinations oflayer_id_ext and nuh_layer_id is completely symmetric and there is nounused bit representation. The inferred value of layer_id_ext may beexploited in the data stream by transmitting layer_id_ext such that thebits spanned in the data stream for the transmission of layer_id_ext isreduced so to account for the fact that layer_id_ext in fact merelyassumes three different values {1, 2, 3}. For example, a binary unarycode may be used to map the three elements of {1, 2, 3} onto binarystrings which are then, for example, encoded into the data stream, anddecoded therefrom, using entropy coding/decoding.

Thus, inter alias, it has been described with respect to the latterconcept, a device configured to process a multi-layered data stream intowhich a video material is coded at different layers, corresponding todifferent levels of information amount, using inter-layer prediction,the multi-layer data stream comprising a plurality of packets, each ofwhich is associated with one of the different layers, the deviceconfigured to, for each of the packets of the multi-layered data stream,read a base layer-ID field from the multi-layered data stream, the baselayer-ID field comprising a first sub-field and a second sub-field;check as to whether the first sub-field of the base layer-ID fieldfulfills a predetermined criterion, and

-   -   if the first sub-field of the base layer-ID field fulfills the        predetermined criterion, read an extension layer-ID field from        the multi-layered data stream, derive an extension value using        the extension layer-ID field so that the extension value lies        within a first subset of a domain of extension values, and        derive a cluster value from the second sub-field by mapping from        a domain of possible values of the second sub-field onto a        domain of cluster values; and    -   if the first sub-field of the base layer-ID field does not        fulfill the predetermined criterion, refrain from reading an        extension layer-ID field from the multi-layered data stream, set        the extension value to a value disjoint to the first subset of        the domain of extension values, and derive the cluster value        from the second sub-field by mapping from the domain of possible        values of the second sub-field onto the domain of cluster        values,

wherein the device indexes the layer which the respective packet isassociated with using the cluster and extension values. It should benoted that the predetermined criterion may be read from the multi-layerdata stream. As described above, “first sub-field” may be one bit longand, in particular, LSB; “predetermined criterion” may be “if one”,“domain of extension values” may be, for example, {0,1,2,3}, “firstsubset” may be {1,2,3}, “using the extension layer-ID field” may be aunitary matrix applied to extension layer-ID field, alternatively:adding the value of the first sub-field to the result of the unitarymatrix application, “set the extension value to a value disjoint to” maybe a setting “to 0”; “mapping from the domain of possible values of thefirst sub-field onto the domain of cluster values” may be the same forboth cases, but may different alternatively; the mapping may bebijective, and “index the layer which the respective packet isassociated with using the cluster and extension values” may involveusing, for example, the cluster value for setting a, more significantdigit and the extension value to set a lower significant digit.

The latter concept directly translates into an encoder configured toencode into a multi-layered data stream 204 a video material 202 atdifferent layers, corresponding to different levels of informationamount, using inter-layer prediction, the multi-layer data streamcomprising a plurality of packets 206, each of which is associated withone of the different layers, wherein the layer which the respectivepacket is associated with, is uniquely determined by a cluster andextension value, the encoder configured to, for each for the packets 206of the multi-layered data stream, if the extension value is within afirst subset of a domain of extension values, insert a base layer-IDfield 308 into the multi-layered data stream, the base layer-ID fieldcomprising a first sub-field 308 a and a second sub-field 308 b, withusing the cluster value to set the second sub-field and setting thefirst sub-field so as to fulfill a predetermined criterion; and insertan extension layer-ID into the multi-layered data stream, with using theextension value to set the extension layer-ID; and, if the extensionvalue equals a value disjoint to the first subset of the domain ofextension values, refrain from inserting an extension layer ID field andinsert 312 a base layer-ID field 308 into the multi-layered data stream,the base layer-ID field comprising a first sub-field 308 a and a secondsub-field 308 b, with using the cluster value to set the secondsub-field and setting the first sub-field so as to not fulfill thepredetermined criterion. That is, the encoder generates the data streamso that the decoder may derive the correct layer-ID in the manneroutlined above and uses to this end inverse mappings compared to thoseoutlined above with respect to the decoding or receiving side.

Summarizing above description, same provided an extension mechanism forthe layer identifier of enhancement layer data, the grouping of layeridentifiers into clusters and an efficient signaling of codingdependencies within clusters.

The details set out above may be apply to the data of enhancement layersin a layered coding scenario, while data of the base layer is keptunchanged. The base layer of the layered coding scenario may beidentified through a layer identifier (cp. nuh_layer_id) in the commonheader syntax of base and enhancement layer data packets.

In the above examples, the respective device at the recipients side maybe a video decoder. The video decoder, thus operating, would be able toselect packets of the bitstream identified by (the extended) layerID fordecoding. The device could, however, alternatively be a network elementwhich would be able to discard packets of a bitstream that areidentified by (the extended) layerID based on external factors such asnetwork utilization, knowledge about decoders, etc.

In all of the above embodiments, the multi-layered data stream may havethe video material coded there into at the different layers usinginter-layer prediction such that any layer inter-layer predicted fromanother layer adds one or more of further views; depth information;surface reflectance information; color component information; spatialresolution refinement; and SNR resolution refinement.

Although some aspects have been described in the context of anapparatus, it is clear that these aspects also represent a descriptionof the corresponding method, where a block or device corresponds to amethod step or a feature of a method step. Analogously, aspectsdescribed in the context of a method step also represent a descriptionof a corresponding block or item or feature of a correspondingapparatus. Some or all of the method steps may be executed by (or using)a hardware apparatus, like for example, a microprocessor, a programmablecomputer or an electronic circuit. In some embodiments, some one or moreof the most important method steps may be executed by such an apparatus.

The inventive encoded signals mentioned above can be stored on a digitalstorage medium or can be transmitted on a transmission medium such as awireless transmission medium or a wired transmission medium such as theInternet.

Depending on certain implementation requirements, embodiments of theinvention can be implemented in hardware or in software. Theimplementation can be performed using a digital storage medium, forexample a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM,an EEPROM or a FLASH memory, having electronically readable controlsignals stored thereon, which cooperate (or are capable of cooperating)with a programmable computer system such that the respective method isperformed. Therefore, the digital storage medium may be computerreadable.

Some embodiments according to the invention comprise a data carrierhaving electronically readable control signals, which are capable ofcooperating with a programmable computer system, such that one of themethods described herein is performed.

Generally, embodiments of the present invention can be implemented as acomputer program product with a program code, the program code beingoperative for performing one of the methods when the computer programproduct runs on a computer. The program code may for example be storedon a machine readable carrier.

Other embodiments comprise the computer program for performing one ofthe methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, acomputer program having a program code for performing one of the methodsdescribed herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a datacarrier (or a digital storage medium, or a computer-readable medium)comprising, recorded thereon, the computer program for performing one ofthe methods described herein. The data carrier, the digital storagemedium or the recorded medium are typically tangible and/ornon-transitionary.

A further embodiment of the inventive method is, therefore, a datastream or a sequence of signals representing the computer program forperforming one of the methods described herein. The data stream or thesequence of signals may for example be configured to be transferred viaa data communication connection, for example via the Internet.

A further embodiment comprises a processing means, for example acomputer, or a programmable logic device, configured to or adapted toperform one of the methods described herein.

A further embodiment comprises a computer having installed thereon thecomputer program for performing one of the methods described herein.

A further embodiment according to the invention comprises an apparatusor a system configured to transfer (for example, electronically oroptically) a computer program for performing one of the methodsdescribed herein to a receiver. The receiver may, for example, be acomputer, a mobile device, a memory device or the like. The apparatus orsystem may, for example, comprise a file server for transferring thecomputer program to the receiver.

In some embodiments, a programmable logic device (for example a fieldprogrammable gate array) may be used to perform some or all of thefunctionalities of the methods described herein. In some embodiments, afield programmable gate array may cooperate with a microprocessor inorder to perform one of the methods described herein. Generally, themethods are advantageously performed by any hardware apparatus.

While this invention has been described in terms of several embodiments,there are alterations, permutations, and equivalents which fall withinthe scope of this invention. It should also be noted that there are manyalternative ways of implementing the methods and compositions of thepresent invention. It is therefore intended that the following appendedclaims be interpreted as including all such alterations, permutationsand equivalents as fall within the true spirit and scope of the presentinvention.

REFERENCES

-   [1] B. Bross, W.-J. Han, J.-R. Ohm, G. J. Sullivan, T. Wiegand    (Eds.), “High Efficiency Video Coding (HEVC) text specification    draft 10”, JCTVC-L1003, Geneva, CH, January 2013-   [2] G. Tech, K. Wegner, Y. Chen, M. Hannuksela, J. Boyce (Eds.),    “MV-HEVC Draft Text 3 (ISO/IEC 23008-2 PDAM2)”, JCT3V-C1004, Geneva,    CH, January 2013-   [3] G. Tech, K. Wegner, Y. Chen, S. Yea (Eds.), “3D-HEVC Test Model    Description, draft specification”, JCT3V-C1005, Geneva, CH, January    2013-   [4] WILBURN, Bennett, et al. High performance imaging using large    camera arrays. ACM Transactions on Graphics, 2005, 24. Jg., Nr.    3, S. 765-776.-   [5] WILBURN, Bennett S., et al. Light field video camera. In:    Electronic Imaging 2002. International Society for Optics and    Photonics, 2001. S. 29-36.-   [6] HORIMAI, Hideyoshi, et al. Full-color 3D display system with 360    degree horizontal viewing angle. In: Proc. Int. Symposium of 3D and    Contents. 2010. S. 7-10.

The invention claimed is:
 1. A device configured to process a datastream, representing an encoded video signal including a plurality ofsets of video signals, each set of video signals including a multilayervideo signal, the device being configured to: read, from the datastream, a first base layer-ID associated with a first set of videosignals, the first base layer-ID being common for each layer of themultilayer video signal of the first set of video signals, a firstinter-dependency syntax structure indicating an inter-dependency betweenlayers of the multilayer video signal of the first set of video signals,and a second inter-dependency syntax structure indicating aninter-dependency between a layer of the multilayer video signal of thefirst set of video signals and a layer of the multilayer video signal ofa second set of video signals; and decode the encoded video signal basedon the first inter-dependency syntax structure and the secondinter-dependency syntax structure, wherein the interdependency betweenlayers of the multilayer video signal is based at least partially on aspatial relation or a coding dependency among layers of the multilayervideo signal of the first set of video signals or between the layer ofthe multilayer video signal of the first set of video signals and thelayer of the multilayer video signal of the second set of video signals.2. The device of claim 1, wherein the second inter-dependency syntaxstructure indicates whether the second set of video signals is a directreference for the first set of video signals.
 3. The device of claim 1,the device being configured to read, from the data stream, a layer-IDassociated with each layer of the multilayer video signal of the firstset of video signals.
 4. The device of claim 1, wherein the multilayervideo signal of at least one of the plurality of sets of video signalsincludes different layers encoded thereinto using inter-layer predictionsuch that any layer of the multilayer video signal that is inter-layerpredicted from another layer of the multilayer video signal adds atleast one of: further views; depth information; alpha blendinginformation; color component information; spatial resolution refinement;and SNR resolution refinement.
 5. The device of claim 1, where the firstinter-dependency syntax structure indicates interdependencies betweendifferent sets of layers of the multilayer video signal of the first setof video signals, each of the different sets of layers being associatedwith the first base layer-ID.
 6. The device of claim 1, wherein thedevice is configured to: read, from the data stream, a plurality ofinstantiations of the second inter-dependency syntax structure fordifferent pairs (j,k) of values 0<k≤j≤n<N representable by a baselayer-ID, with N being the number of values representable by the baselayer-ID, by reading, or suppressing reading, the secondinter-dependency structure from the data stream for pair (j,k)conditionally depending on the first inter-dependency syntax structureindicating a presence or non-presence of interdependency between pair(j,k), wherein one of the plurality of instantiations of the secondinter-dependency syntax structure read for a pair (j,k) with j=k,indicates which layers among those comprising base layer-ID j areinter-layer prediction dependent on other layers among those comprisingbase layer-ID j, and another of the plurality of instantiations of thesecond inter-dependency structure read for pair (j,k), with k<jindicates which layers among those comprising base layer-ID j areinter-layer prediction dependent on layers among those comprising baselayer-ID k.
 7. The device of claim 1, wherein the device is configuredto: read instantiations of the second inter-dependency syntax structurefrom the data stream, along with, for each instantiation of the secondinter-dependency syntax structure, an association syntax structure whichassociates the respective instantiation of the second inter-dependencysyntax structure with a respective subset of pairs (j,k) of values0<k≤j≤n<N representable by a base layer-ID, with N being the number ofvalues representable by the base layer-ID.
 8. An encoder configured toencode a video into a plurality of sets of video signals such that eachset of video signals includes a multilayer video signal, the encoderbeing configured to: insert, into a data stream representing an encodedvideo, a first base layer-ID associated with a first set of videosignals, the first base layer-ID being common for each layer of themultilayer video signal of the first set of video signals; a firstinter-dependency syntax structure indicating an inter-dependency betweenlayers of the multilayer video signal of the first set of video signals;and a second inter-dependency syntax structure indicating aninter-dependency between a layer of the multilayer video signal of thefirst set of video signals and a layer of the multilayer video signal ofa second set of video signals, wherein the encoded video is to bedecoded based on the first inter-dependency syntax structure and thesecond inter-dependency syntax structure, and the interdependencybetween layers of the multilayer video signal is based at leastpartially on a spatial relation or a coding dependency among layers ofthe multilayer video signal of the first set of video signals or betweenthe layer of the multilayer video signal of the first set of videosignals and the layer of the multilayer video signal of the second setof video signals.
 9. The encoder of claim 8, wherein the secondinter-dependency syntax structure indicates whether the second set ofvideo signals is a direct reference for the first set of video signals.10. The encoder of claim 8, the device being configured to insert, intothe data stream, a layer-ID associated with each layer of the multilayervideo signal of the first set of video signals.
 11. The encoder of claim8, wherein the multilayer video signal of at least one of the pluralityof sets of video signals includes different layers encoded thereintousing inter-layer prediction such that any layer of the multilayer videosignal that is inter-layer predicted from another layer of themultilayer video signal adds at least one of: further views; depthinformation; alpha blending information; color component information;spatial resolution refinement; and SNR resolution refinement.
 12. Theencoder of claim 8, where the first inter-dependency syntax structureindicates interdependencies between different sets of layers of themultilayer video signal of the first set of video signals, each of thedifferent sets of layers being associated with the first base layer-ID.13. The encoder of claim 8, wherein the device is configured to: insert,into the data stream, a plurality of instantiations of the secondinter-dependency syntax structure for different pairs (j,k) of values0<k≤j≤n<N representable by a base layer-ID, with N being the number ofvalues representable by the base layer-ID, by inserting, or suppressinginserting, the second inter-dependency structure into the data streamfor pair (j,k) conditionally depending on the first inter-dependencysyntax structure indicating a presence or non-presence ofinterdependency between pair (j,k), wherein one of the plurality ofinstantiations of the second inter-dependency syntax structure insertedfor a pair (j,k) with j=k, indicates which layers among those comprisingbase layer-ID j are inter-layer prediction dependent on other layersamong those comprising base layer-ID j, and another of the plurality ofinstantiations of the second inter-dependency structure inserted forpair (j,k), with k<j indicates which layers among those comprising baselayer-ID j are inter-layer prediction dependent on layers among thosecomprising base layer-ID k.
 14. The encoder of claim 8, wherein thedevice is configured to: insert instantiations of the secondinter-dependency syntax structure into the data stream, along with, foreach instantiation of the second inter-dependency syntax structure, anassociation syntax structure which associates the respectiveinstantiation of the second inter-dependency syntax structure with arespective subset of pairs (j,k) of values 0<k≤j≤n<N representable by abase layer-ID, with N being the number of values representable by thebase layer-ID.
 15. A non-transitory computer-readable medium for storingdata associated with a video, comprising a data stream stored in thenon-transitory computer-readable medium and representing an encodedvideo signal including a plurality of sets of video signals, each set ofvideo signals including a multilayer video signal, the data streamcomprising: a first base layer-ID associated with a first set of videosignals, the first base layer-ID being common for each layer of themultilayer video signal of the first set of video signals; a firstinter-dependency syntax structure indicating an inter-dependency betweenlayers of the multilayer video signal of the first set of video signals;and a second inter-dependency syntax structure indicating aninter-dependency between a layer of the multilayer video signal of thefirst set of video signals and a layer of the multilayer video signal ofa second set of video signals, wherein the encoded video is to bedecoded based on the first inter-dependency syntax structure and thesecond inter-dependency syntax structure, and the interdependencybetween layers of the multilayer video signal is based at leastpartially on a spatial relation or a coding dependency among layers ofthe multilayer video signal of the first set of video signals or betweenthe layer of the multilayer video signal of the first set of videosignals and the layer of the multilayer video signal of the second setof video signals.
 16. The non-transitory computer-readable medium ofclaim 15, wherein the second inter-dependency syntax structure indicateswhether the second set of video signals is a direct reference for thefirst set of video signals.
 17. The non-transitory computer-readablemedium of claim 15, the data stream further includes a layer-IDassociated with each layer of the multilayer video signal of the firstset of video signals.
 18. The non-transitory computer-readable medium ofclaim 15, wherein the multilayer video signal of at least one of theplurality of sets of video signals includes different layers encodedthereinto using inter-layer prediction such that any layer of themultilayer video signal that is inter-layer predicted from another layerof the multilayer video signal adds at least one of: further views;depth information; alpha blending information; color componentinformation; spatial resolution refinement; and SNR resolutionrefinement.
 19. The non-transitory computer-readable medium of claim 15,where the first inter-dependency syntax structure indicatesinterdependencies between different sets of layers of the multilayervideo signal of the first set of video signals, each of the differentsets of layers being associated with the first base layer-ID.